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METHOD AND APPARATUS FOR CONTROLLING LIGHTING 
BASED ON USER BEHAVIOR 



Field of the Invention 

The present invention relates to methods and apparatus 
for controlling lighting devices^ and more particularly, to a 
method and apparatus for automatically controlling lighting 
devices based on user activity. 



Background of the Invention 

The consumer marketplace offers a number of lighting 
devices and lighting controllers that include features intended 
to increase the convenience and capabilities of these devices. 

15 Many available lighting devices, for example, have an associated 
remote control device that allows the user to adjust a number of 
the light settings remotely. For example, a user can typically 
activate a light or adjust the intensity, direction or other 
settings of the light using the remote control. The Clapper™ 

20 lighting controller, for example, controls lights or other 
appliances by detecting the sound of clapping hands. In one 
variation, the Clapper™ lighting controller can control multiple 
lights or appliances, with each device assigned a unique number 
of claps to control the device. 

25 While such remote controls have greatly improved the 

convenience of lighting devices, they still require the 
affirmative action of the user to manipulate the remote control 
(or another input mechanism associated with the device) to turn 
on the light or to indicate the manner in which the light should 

30 be adjusted. Thus, if the remote control is not readily 
available, or the user does not wish to move closer to the device 
itself, the user may still be unable to conveniently activate the 
lighting device or adjust one or more light settings in a desired 
manner . 
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It has been observed that there is often a predictable 
relationship between certain user activity and a corresponding 
manner in which the settings of a light should be adjusted. For 
example, it may be observed that when a particular user sits in a 
5 particular chair at a certain time of day, the user also tends to 
turn on a certain light using tihe remote control. There is 
currently no mechanism, however, that learns such ritualistic 
user activity and automatically activates or adjusts a lighting 
device . 

10 A need therefore exists for a lighting controller that 

monitors user activity and automatically activates or adjusts a 
light in response to predefined events or behavior. A further 
need exists for a lighting controller that employs a rule-base to 
define user activities or events, as well as the corresponding 

15 responses that should be implemented to activate or adjust the 
lighting device. 

Summary of the Invention 

Generally, a method and apparatus are disclosed for 
20 automatically controlling a lighting device based on user 
activity. The disclosed lighting controller processes at least 
one of audio and video information to identify predefined user 
events that suggest that a lighting device should be activated or 
adjusted. 

25 According to an exemplary rule-based implementation of 

the invention, a number of behavior rules can define various user 
activities or events that suggest that the user would like to 
activate or adjust a lighting device. Each rule contains one or 
more conditions, and, optionally, a corresponding action-item 

30 that should be performed when the rule is satisfied to activate 
or adjust the lighting device. Upon detection of a predefined 
event, the corresponding action, if any, is performed by the 
lighting controller. 
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A more complete understanding of the present invention, 
as well as further features and advantages of the present 
invention, will be obtained by reference to the following 
detailed description and drawings. 

5 

Brief Description of the Drawings — 

FIG. 1 illustrates a lighting controller in accordance 
with the present invention; 

FIG. 2 illustrates a sample table from the user profile 
10 of FIG. 1 in accordance with the present invention; and 

FIG. 3 is a flow chart describing an exemplary user 
event monitoring process embodying principles of the present 
invention . 

15 Detailed Description 

FIG. 1 illustrates a lighting controller 100 in 
accordance with the present invention. As shown in FIG. 1, the 
lighting controller 100 includes one or more audio/visual capture 
devices 150-1 through 150-N (hereinafter, collectively referred 

20 to as audio/visual capture devices 150) that are focused on one 
or more monitored areas 140. 

Each audio/visual capture device 150 may be embodied, 
for example, as a fixed or pan-tilt-zoom (PTZ) camera for 
capturing image or video information, or one or more microphones 

25 for capturing audio information (or both) . The audio and video 
information generated by the audio/visual capture devices 150 are 
processed by the lighting controller 100, in a manner discussed 
below in conjunction with FIG. 3, to identify one or more 
predefined user activities or events suggesting that the user 

30 would like to activate or adjust one or more lighting device (s) 
160. In a rule-based implementation, the present invention 
employs a user profile 200, discussed further below in 
conjunction with FIG. 2, that records a number of behavior rules 
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identifying user behavior that should initiate the activation or 
adjustment of a light 160 in a specified manner. 

The user behavior defined in the profile 200 may be 
detected by the lighting controller 100 in accordance with the 
5 present invention. As discussed further below, each exemplary 
behavior rule contains one or more criteria that must be 
satisfied in order for the rule to be triggered, and, optionally, 
a corresponding action-item that should be performed by the 
lighting controller 100 to activate or adjust a light 160 when 
10 the predefined criteria for initiating the behavior rule is 
satisfied. 

At least one of the criteria for each rule is a 
condition detected in the audio or video information generated by 
the audio/visual capture devices 150 using audio or vision-based 

15 techniques, in accordance with the present invention. Upon 
detection of such predefined user behavior, the corresponding 
action, if any, is performed by the lighting controller 100. 
Typically, the corresponding action is the issuance of a command 
to adjust or activate (turn on or off) the light 160. 

20 As shown in FIG. 1, and discussed further below in 

conjunction with FIG. 3, the lighting controller 100 also 
contains a user event monitoring process 300. Generally, the 
user event monitoring process 3 00 processes the audio information 
or images obtained by the audio /visual capture devices 150 and 

25 detects one or more predefined behavior actions that should 
trigger the activation or adjustment of a light 160. 

The lighting controller 100 may be embodied as any 
computing device, such as a personal computer or workstation, 
that contains a processor 120, such as a central processing unit 

30 (CPU), and memory 110, such as RAM and/or ROM. Alternatively, 
the lighting controller 100 may be embodied as an application 
specific integrated circuit (ASIC) (not shown) that is included, 




-4- 



) 



US010634 



for example, in a television, set-top terminal or another 
electronic device. 

FIG. 2 illustrates an exemplary table of the user 
profile (s) 2 00 that records various behavioral rules for one or 
5 more users. Each rule in the user profile (s) 200 identifies the 
corresponding user(s) and includes predefined criteria specifying 
the conditions under which the rule should be initiated, and, 
optionally, a corresponding action item that should be triggered 
when the criteria associated with the rule is satisfied. 

10 Typically, the action item identifies the light (s) 160 that 
should be activated or adjusted when the rule is triggered. 

As shown in FIG. 2, the user profile 200 is comprised 
of a plurality of records, such as records 205-208, each 
associated with a different behavioral rule. For each rule, the 

15 user profile 200 identifies the corresponding user(s) in field 
250, the corresponding rule criteria for a given behavioral event 
in field 260 and the corresponding action, if any, in field 270.. 

For example, the user habits recorded in record 205 for 
the user, John Smith, indicates that the user generally turns on 

20 the light after checking the mail and sitting in a certain chair. 
Likewise, the user habits recorded in record 2 06 for the user, 
Jane Smith, indicates that the user likes to read the newspaper 
while on the couch. Finally, the exemplary user habits recorded 
in record 208 for all users indicates that most users would like 

25 to have a light on when they sit in a certain seat with their 
feet up. The corresponding action item associated with each rule 
typically activates a light with an appropriate intensity and/or 
direction (variable intensity and position) or otherwise adjusts 
the intensity, direction or other settings of a lighting device. 

30 Generally, the user behavior recorded in the user 

profile(s) 200 can be obtained explicitly, i.e., from survey 
responses, or implicitly, by monitoring how a given user responds 
to a given set of circumstances. Thereafter, a rule can be 
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established that defines the given set of circumstances and the 
corresponding action item that should be performed. 

FIG. 3 is a flow chart describing an exemplary user 
event monitoring process 300. The user event monitoring process 
5 300 processes audio or video information (or both) obtained from i 

the audio/visual capture devices 150 and detects one or more user ! 

i 

events defined in the user profile 200. The exemplary user event I 
monitoring process 300 is a general process illustrating the 
broad concepts of the present invention. As shown in FIG. 3, the 

10 user event monitoring process 300 initially obtains one or more 
inputs from the audio/visual capture devices 150 during step 305. 
Thereafter, the user event monitoring process 3 00 optionally 
identifies the user(s) .that are present during step 310, for 
example, using a biometric evaluation of the audio or visual 

15 information obtained from the audio/visual capture device 150. 

Thereafter, the audio/visual information is analyzed 
during step 320 using audio and/or video content analysis (VCA) 
techniques. For a detailed discussion of suitable audio content 
analysis techniques, see, for example, Silvia Pfeiffer et al., 

20 "Automatic Audio Content Analysis," Proc . ACM Multimedia 96, 21- 
30, Boston, MA. (Nov. 1996), incorporated by reference herein. 
For a detailed discussion of suitable VCA techniques, see, for 
example, Nathanael Rota and Monique Thonnat, "Video Sequence 
Interpretation for Visual Surveillance," in Proc. of the 3d IEEE 

25 Int'l Workshop on Visual Surveillance, 59- 67, Dublin, Ireland 
(July 1, 2000), and Jonathan Owens and Andrew Hunter, 
"Application of the Self -Organizing Map to Trajectory 
Classification,' in Proc. of the 3d IEEE Int'l Workshop on Visual 
Surveillance, 77-83, Dublin, Ireland (July 1, 2000), incorporated 

30 by reference herein. Generally, the audio content analysis and 
VCA techniques are employed to recognize various features in the 
signals obtained by the audio/visual capture devices 150. 
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A test is performed during step 33 0 to determine if the 
audio/video content analysis detects a predefined event, as 
defined in the user profile 300* If it is determined during step 
330 that the audio/video content analysis does not detect a 
5 predefined event, then program control returns to step 310 to 
continue monitoring user activities^in the manner discussed 
above . 

If, however, it is determined during step 3 30 that the 
audio/video content analysis detects a predefined user event, 
10 then the event is processed during step 340 as indicated in field 
270 of the user profile 200, if any, for the identified user. 
Program control then terminates (or returns to step 310 and 
continues monitoring user activities in the manner discussed 
above) . 

15 It is to be understood that the embodiments and 

variations shown and described herein are merely illustrative of 
the principles of this invention and that various modifications 
may be implemented by those skilled in the art without departing 
from the scope and spirit of the invention. 

20 
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